Model-based Free-viewpoint Video: Acquisition, Rendering, and Encoding

In recent years, the convergence of computer vision and computer graphics has put forth free-viewpoint video as a new field of research. The goal is to advance traditional 2D video into an immersive medium that enables the viewer to interactively choose an arbitrary viewpoint in 3D space onto the a scene while it plays back. In this paper we give an overview of a system for reconstructing, rendering and encoding free-viewpoint videos of human actors. It employs a hardware-accelerated marker-free optical motion capture algorithm from multi-view video streams and an a-priori body model to reconstruct shape and motion of a moving actor. Real-time high-quality rendering of the moving person from arbitrary perspectives is achieved by applying a multi-view texturing approach from the video frames. We also present a predictive encoding as well as a 4D-SPIHT wavelet compression mechanism that both exploit the 3D scene geometry for efficient encoding of the multi-view texture images.